Community based complex interventions to sustain independence in older people: systematic review and network meta-analysis.

OBJECTIVE: To synthesise evidence of the effectiveness of community based complex interventions, grouped according to their intervention components, to sustain independence for older people. DESIGN: Systematic review and network meta-analysis. DATA SOURCES: Medline, Embase, CINAHL, PsycINFO, CENTRAL, clinicaltrials.gov, and International Clinical Trials Registry Platform from inception to 9 August 2021 and reference lists of included studies. ELIGIBILITY CRITERIA: Randomised controlled trials or cluster randomised controlled trials with ≥24 weeks' follow-up studying community based complex interventions for sustaining independence in older people (mean age ≥65 years) living at home, with usual care, placebo, or another complex intervention as comparators. MAIN OUTCOMES: Living at home, activities of daily living (personal/instrumental), care home placement, and service/economic outcomes at 12 months. DATA SYNTHESIS: Interventions were grouped according to a specifically developed typology. Random effects network meta-analysis estimated comparative effects; Cochrane's revised tool (RoB 2) structured risk of bias assessment. Grading of recommendations assessment, development and evaluation (GRADE) network meta-analysis structured certainty assessment. RESULTS: The review included 129 studies (74 946 participants). Nineteen intervention components, including "multifactorial action from individualised care planning" (a process of multidomain assessment and management leading to tailored actions), were identified in 63 combinations. For living at home, compared with no intervention/placebo, evidence favoured multifactorial action from individualised care planning including medication review and regular follow-ups (routine review) (odds ratio 1.22, 95% confidence interval 0.93 to 1.59; moderate certainty); multifactorial action from individualised care planning including medication review without regular follow-ups (2.55, 0.61 to 10.60; low certainty); combined cognitive training, medication review, nutritional support, and exercise (1.93, 0.79 to 4.77; low certainty); and combined activities of daily living training, nutritional support, and exercise (1.79, 0.67 to 4.76; low certainty). Risk screening or the addition of education and self-management strategies to multifactorial action from individualised care planning and routine review with medication review may reduce odds of living at home. For instrumental activities of daily living, evidence favoured multifactorial action from individualised care planning and routine review with medication review (standardised mean difference 0.11, 95% confidence interval 0.00 to 0.21; moderate certainty). Two interventions may reduce instrumental activities of daily living: combined activities of daily living training, aids, and exercise; and combined activities of daily living training, aids, education, exercise, and multifactorial action from individualised care planning and routine review with medication review and self-management strategies. For personal activities of daily living, evidence favoured combined exercise, multifactorial action from individualised care planning, and routine review with medication review and self-management strategies (0.16, -0.51 to 0.82; low certainty). For homecare recipients, evidence favoured addition of multifactorial action from individualised care planning and routine review with medication review (0.60, 0.32 to 0.88; low certainty). High risk of bias and imprecise estimates meant that most evidence was low or very low certainty. Few studies contributed to each comparison, impeding evaluation of inconsistency and frailty. CONCLUSIONS: The intervention most likely to sustain independence is individualised care planning including medicines optimisation and regular follow-up reviews resulting in multifactorial action. Homecare recipients may particularly benefit from this intervention. Unexpectedly, some combinations may reduce independence. Further research is needed to investigate which combinations of interventions work best for different participants and contexts. REGISTRATION: PROSPERO CRD42019162195.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed.

OBJECTIVES: Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement. STUDY DESIGN AND SETTING: We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs. RESULTS: Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD. CONCLUSION: Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time.

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

Two-stage or not two-stage? That is the question for IPD meta-analysis projects.

Individual participant data meta-analysis (IPDMA) projects obtain, check, harmonise and synthesise raw data from multiple studies. When undertaking the meta-analysis, researchers must decide between a two-stage or a one-stage approach. In a two-stage approach, the IPD are first analysed separately within each study to obtain aggregate data (e.g., treatment effect estimates and standard errors); then, in the second stage, these aggregate data are combined in a standard meta-analysis model (e.g., common-effect or random-effects). In a one-stage approach, the IPD from all studies are analysed in a single step using an appropriate model that accounts for clustering of participants within studies and, potentially, between-study heterogeneity (e.g., a general or generalised linear mixed model). The best approach to take is debated in the literature, and so here we provide clearer guidance for a broad audience. Both approaches are important tools for IPDMA researchers and neither are a panacea. If most studies in the IPDMA are small (few participants or events), a one-stage approach is recommended due to using a more exact likelihood. However, in other situations, researchers can choose either approach, carefully following best practice. Some previous claims recommending to always use a one-stage approach are misleading, and the two-stage approach will often suffice for most researchers. When differences do arise between the two approaches, often it is caused by researchers using different modelling assumptions or estimation methods, rather than using one or two stages per se.

Predicting antipsychotic-induced weight gain in first episode psychosis - A field-wide systematic review and meta-analysis of non-genetic prognostic factors.

BACKGROUND: Whether non-genetic prognostic factors significantly influence the variable prognosis of antipsychotic-induced weight gain (AIWG) has not yet been systematically explored. METHODS: Searches for both randomized and non-randomized studies were undertaken using four electronic databases, two trial registers, and via supplemental searching methods. Unadjusted and adjusted estimates were extracted. Meta-analyses were undertaken using a random-effects generic inverse model. Risk of bias and quality assessments were undertaken using Quality in Prognosis Studies (QUIPS) and Grading of Recommendations Assessment, Development and Evaluation (GRADE), respectively. RESULTS: Seventy-two prognostic factors were assessed across 27 studies involving 4426 participants. Only age, baseline body mass index (BMI), and sex were suitable for meta-analysis. Age (b=-0.044, 95%CI -0.157-0.069), sex (b=0.236, 95%CI -0.086-0.558), and baseline BMI (b=-0.013 95%CI -0.225-0.200) were associated with nonsignificant effects on AIWG prognosis. The highest quality GRADE rating was moderate in support of age, trend of early BMI increase, antipsychotic treatment response, unemployment, and antipsychotic plasma concentration. Trend of early BMI increase was identified as the most clinically significant prognostic factor influencing long-term AIWG prognosis. CONCLUSIONS: The strong prognostic information provided by BMI trend change within 12 weeks of antipsychotic initiation should be included within AIWG management guidance to highlight those at highest risk of worse long-term prognosis. Antipsychotic switching and resource-intensive lifestyle interventions should be targeted toward this cohort. Our results challenge previous research that several clinical variables significantly influence AIWG prognosis. We provide the first mapping and statistical synthesis of studies examining non-genetic prognostic factors of AIWG and highlight practice, policy, and research implications.

Risk-of-bias assessment using Cochrane's revised tool for randomized trials (RoB 2) was useful but challenging and resource-intensive: observations from a systematic review.

OBJECTIVES: To report our experience using version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2). STUDY DESIGN AND SETTING: Two reviewers independently applied RoB 2 to results of interest in a large systematic review of complex interventions and reached consensus. We recorded the time taken, and noted and discussed our difficulties using the tool, and the resolutions we adopted. We explored the time taken with regression analysis and summarized our experience of implementing the tool. RESULTS: We assessed risk of bias in 860 results of interest in 113 studies. Staff resource averaged 358 minutes per study (SD 183). Number of results (ß = 22) and reports (ß = 14) per study and experience of the team (ß = -6) significantly affected assessment time. To implement the tool consistently, we developed cut points for missingness and considerations of balance regarding missingness, assumed some concerns with intervention deviations unless otherwise prevented or investigated, some concerns with measurements from unblinded self-reporting participants, and judged low risk of selection for certain dichotomous outcomes despite the absence of an analysis plan. CONCLUSION: The RoB 2 tool and guidance are useful but resource-intensive and challenging to implement. Critical appraisal tools and reporting guidelines should detail risk of bias implementation. Improved guidance focusing on implementation could assist reviewers.

Calculating the power of a planned individual participant data meta-analysis of randomised trials to examine a treatment-covariate interaction with a time-to-event outcome.

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers should consider the power of their planned IPDMA conditional on the studies promising their IPD and their characteristics. Such power estimates help inform whether the IPDMA project is worth the time and funding investment, before IPD are collected. Here, we suggest how to estimate the power of a planned IPDMA of randomised trials aiming to examine treatment-covariate interactions at the participant-level (i.e., treatment effect modifiers). We focus on a time-to-event (survival) outcome with a binary or continuous covariate, and propose an approximate analytic power calculation that conditions on the actual characteristics of trials, for example, in terms of sample sizes and covariate distributions. The proposed method has five steps: (i) extracting the following aggregate data for each group in each trial-the number of participants and events, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate; (ii) specifying a minimally important interaction size; (iii) deriving an approximate estimate of Fisher's information matrix for each trial and the corresponding variance of the interaction estimate per trial, based on assuming an exponential survival distribution; (iv) deriving the estimated variance of the summary interaction estimate from the planned IPDMA, under a common-effect assumption, and (v) calculating the power of the IPDMA based on a two-sided Wald test. Stata and R code are provided and a real example provided for illustration. Further evaluation in real examples and simulations is needed.

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression.

AIMS: Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n) is appropriate relative to the number of events (Ek) and the number of predictor parameters (pk) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes. PROPOSED CRITERIA: The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R2 of the multinomial logistic regression. EVALUATION OF CRITERIA: We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation. SUMMARY: We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

Calculating the power to examine treatment-covariate interactions when planning an individual participant data meta-analysis of randomized trials with a binary outcome.

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers and funders need assurance it is worth their time and cost. This should include consideration of how many studies are promising their IPD and, given the characteristics of these studies, the power of an IPDMA including them. Here, we show how to estimate the power of a planned IPDMA of randomized trials to examine treatment-covariate interactions at the participant level (ie, treatment effect modifiers). We focus on a binary outcome with binary or continuous covariates, and propose a three-step approach, which assumes the true interaction size is common to all trials. In step one, the user must specify a minimally important interaction size and, for each trial separately (eg, as obtained from trial publications), the following aggregate data: the number of participants and events in control and treatment groups, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate. This allows the variance of the interaction estimate to be calculated for each trial, using an analytic solution for Fisher's information matrix from a logistic regression model. Step 2 calculates the variance of the summary interaction estimate from the planned IPDMA (equal to the inverse of the sum of the inverse trial variances from step 1), and step 3 calculates the corresponding power based on a two-sided Wald test. Stata and R code are provided, and two examples given for illustration. Extension to allow for between-study heterogeneity is also considered.

Elixhauser outperformed Charlson comorbidity index in prognostic value after ACS: insights from a national registry.

OBJECTIVE: To compare the performance of risk adjustment models using the Elixhauser and Charlson comorbidity scores in predicting in-hospital outcomes of ACS patients from a nationwide administrative database. STUDY DESIGN AND SETTING: All hospitalizations for ACS in the United States between 2004 and 2014 (n = 7,201,900) were retrospectively analyzed. We used ECS and CCI score based on ICD-9 codes to define comorbidity variables. Logistic regression models were fitted to three in-hospital outcomes, including mortality, Major Acute Cardiovascular & Cerebrovascular Events (MACCE) and bleeding. The prognostic values of ECS and CCI after adjusting for known confounders, were compared using the C-statistic, Akaike information criterion (AIC), and Bayesian information criterion (BIC). RESULTS: The statistical performance of models predicting all in-hospital outcomes demonstrated that the ECS had superior prognostic value compared to the CCI, with higher C-statistics and lower AIC and BIC values associated with the former. CONCLUSION: This is the first study that compared the prognostic value of the ECS and CCI scores in predicting multiple ACS outcomes, based on their scoring systems. Better discrimination and goodness of fit was achieved with the Elixhauser method across all in-hospital outcomes studied.

Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome.

Previous articles in Statistics in Medicine describe how to calculate the sample size required for external validation of prediction models with continuous and binary outcomes. The minimum sample size criteria aim to ensure precise estimation of key measures of a model's predictive performance, including measures of calibration, discrimination, and net benefit. Here, we extend the sample size guidance to prediction models with a time-to-event (survival) outcome, to cover external validation in datasets containing censoring. A simulation-based framework is proposed, which calculates the sample size required to target a particular confidence interval width for the calibration slope measuring the agreement between predicted risks (from the model) and observed risks (derived using pseudo-observations to account for censoring) on the log cumulative hazard scale. Precise estimation of calibration curves, discrimination, and net-benefit can also be checked in this framework. The process requires assumptions about the validation population in terms of the (i) distribution of the model's linear predictor and (ii) event and censoring distributions. Existing information can inform this; in particular, the linear predictor distribution can be approximated using the C-index or Royston's D statistic from the model development article, together with the overall event risk. We demonstrate how the approach can be used to calculate the sample size required to validate a prediction model for recurrent venous thromboembolism. Ideally the sample size should ensure precise calibration across the entire range of predicted risks, but must at least ensure adequate precision in regions important for clinical decision-making. Stata and R code are provided.

Comparative effectiveness of treatment options for subacromial shoulder conditions: a systematic review and network meta-analysis.

BACKGROUND: There are currently many treatment options for patients with subacromial shoulder conditions (SSCs). Clinical decision-making regarding the best treatment option is often difficult. This study aims to evaluate the comparative effectiveness of treatment options for relieving pain and improving function in patients with SSCs. METHODS: Eight databases [including MEDLINE, Embase, CINAHL, AMED, PEDro, Cochrane Database of Systematic Reviews and World Health Organization (WHO) International Clinical Trials Registry] were searched from inception until April 2020. Randomised clinical/controlled trials of adult patients investigating the effects of nonsurgical (e.g. corticosteroid injections, therapeutic exercise, shockwave therapy) and surgical treatment for SSCs, compared with each other, placebo, usual care or no treatment, were retrieved. Pairs of reviewers screened studies independently, quality appraised eligible studies using the Cochrane risk of bias tool, extracted and checked data for accuracy. Primary outcomes were pain and disability in the short term (⩽3 months) and long term (⩾6 months). Direct and indirect evidence of treatment effectiveness was synthesised using random-effects network meta-analysis. RESULTS: The review identified 177 eligible trials. Summary estimates (based on 99 trials providing suitable data, 6764 patients, 20 treatment options) showed small to moderate effects for several treatments, but no significant differences on pain or function between many active treatment comparisons. The primary analysis indicated that exercise and laser therapy may provide comparative benefit in terms of both pain and function at different follow-up time-points, with larger effects found for laser in the short term at 2-6 weeks, although direct evidence was provided by one trial only, and for exercise in the longer term [standardised mean difference (SMD) 0.39, 95% confidence interval (CI) 0.18, 0.59 at 3-6 months] compared with control. Sensitivity analyses excluding studies at increased risk of bias confirmed only the comparative effects of exercise as being robust for both pain and function up until 3-month follow-up. CONCLUSION: Current evidence shows small to moderate effect sizes for most treatment options for SSCs. Six treatments had a high probability of being most effective, in the short term, for pain and function [acupuncture, manual therapy, exercise, exercise plus manual therapy, laser therapy and Microcurrent (MENS) (TENS)], but with low certainty for most treatment options. After accounting for risk of bias, there is evidence of moderate certainty for the comparative effects of exercise on function in patients with SSCs. Future large, high-quality pragmatic randomised trials or meta-analyses are needed to better understand whether specific subgroups of patients respond better to some treatments than others.

Prognostic impact of comorbidity measures on outcomes following acute coronary syndrome: A systematic review.

AIM: To identify existing comorbidity measures and summarise their association with acute coronary syndrome (ACS) outcomes. METHODS: We searched published studies from MEDLINE (OVIDSP) and EMBASE from inception to March 2021, studies of the pre-specified conference proceedings from Web of Science since May 2017, and studies included in any relevant systematic reviews. Studies that reported no comorbidity measures, no association of comorbid burden with ACS outcomes, or only used a comorbidity measure as a confounder without further information were excluded. After independent screening by three reviewers, data extraction and risk of bias assessment of each included study was undertaken. Results were narratively synthesised. RESULTS: Of 4166 potentially eligible studies identified, 12 (combined n = 6 885 982 participants) were included. Most studies had a high risk of bias at quality assessment. Six different types of comorbidity measures were identified with the Charlson comorbidity index (CCI) the most widely used measure among studies. Overall, the greater the comorbid burden or the higher comorbidity scores recorded, the greater was the association with the risk of mortality. CONCLUSION: The review summarised different comorbidity measures and reported that higher comorbidity scores were associated with worse ACS outcomes. The CCI is the most widely measure of comorbid burden and shows additive value to clinical risk scores in use.

Minimum sample size for external validation of a clinical prediction model with a binary outcome.

In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.

Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model.

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

Community-based complex interventions to sustain independence in older people, stratified by frailty: a protocol for a systematic review and network meta-analysis.

INTRODUCTION: Maintaining independence is a primary goal of community health and care services for older people, but there is currently insufficient guidance about which services to implement. Therefore, we aim to synthesise evidence on the effectiveness of community-based complex interventions to sustain independence for older people, including the effect of frailty, and group interventions to identify the best configurations. METHODS AND ANALYSIS: Systematic review and network meta-analysis (NMA). We will include randomised controlled trials (RCTs) and cluster RCTs of community-based complex interventions to sustain independence for older people living at home (mean age ≥65 years), compared with usual care or another complex intervention. We will search MEDLINE (1946 to September 2020), Embase (1947 to September 2020), CINAHL (1981 to September 2020), PsycINFO (1806 to September 2020), CENTRAL and clinical trial registries from inception to September 2020, without date/language restrictions, and scan included papers' reference lists. Main outcomes were: living at home, activities of daily living (basic/instrumental), home-care services usage, hospitalisation, care home admission, costs and cost effectiveness. Additional outcomes were: health status, depression, loneliness, falls and mortality. Interventions will be coded, summarised and grouped. An NMA using a multivariate random-effects model for each outcome separately will determine the relative effects of different complex interventions. For each outcome, we will produce summary effect estimates for each pair of treatments in the network, with 95% CI, ranking plots and measures, and the borrowing of strength statistic. Inconsistency will be examined using a 'design-by-treatment interaction' model. We will assess risk of bias (Cochrane tool V.2) and certainty of evidence using the Grading of Recommendations Assessment, Development and Evaluation for NMA approach. ETHICS AND DISSEMINATION: This research will use aggregated, anonymised, published data. Findings will be reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidance. They will be disseminated to policy-makers, commissioners and providers, and via conferences and scientific journals. PROSPERO REGISTRATION NUMBER: CRD42019162195.

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb.

INTRODUCTION: Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach. METHODS: Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility. RESULTS: Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model. CONCLUSION: Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Minimum sample size for external validation of a clinical prediction model with a continuous outcome.

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.